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ABSTRACT 



In March 1995 the Ohio General Assembly enacted a Pilot 
Project Scholarship Program intended to provide a limited number of vouchers 
to allow Cleveland public school students to attend alternative schools, i.e. 
private schools, including those operated by religious organizations, within 
the Cleveland city limits and public schools in surrounding suburban school 
districts. Vouchers were to be available for the first time in the 1996-97 
school year. Voucher winners were to be chosen by lottery, with voucher 
winners accepted by lottery into the alternative schools with preferences 
given for current students and their siblings. The maximum voucher value was 
to be $2,500, with reductions for increasing income above the federal poverty 
line until eligibility ended at 200% of the poverty level. Students, once 
admitted, could continue receiving vouchers through grade 8, subject to 
continued funding by the state. Although about 4,000 families were made some 
kind of voucher offer, many refused. An evaluation was designed to determine 
why so many did not enroll, and the significance this might have for future 
program management and evaluation. The designed evaluation will have a study 
of parent choice and satisfaction, with interviews with at least 25 parents 
(those that enrolled children and those that did not) , and a study of the 
initial effectiveness of the program in terms of student achievement, subject 
to the limitations of available data. The rationale behind and planned 
implementation of both evaluation components are described. (SLD) 
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TOWARD AN EVALUATION DESIGN FOR THE 
CLEVELAND SCHOLARSHIP PROGRAM 

Paul T. Hill and Stephen P. Klein 
November, 1996 

THE PROGRAM STRUCTURE 

In March 1995 the Ohio General Assembly enacted a Pilot Project Scholarship 
Program intended to provide a limited number of vouchers to allow Cleveland 
public school students to attend "alternative schools," i.e. private schools (including 
schools operated by religious organizations) within the Cleveland city limits and 
public schools in surrounding suburban school districts. 1 Vouchers were to be 
available for the first time during the 1996-1997 school year, for Cleveland students 
entering kindergarten through third grade. Though the program was primarily 
intended to benefit children previously attending public schools, the statute allowed 
up to 50% of recipients to be children already attending private schools. 

The same statute also established a program of tutorial assistance grants for 
children attending public schools. This paper, however, focuses on the voucher, or 
"scholarship" program. 

Voucher winners were to be chosen by lottery. Lottery winners then applied 
to an alternative school that had agreed to accept voucher students. Admissions to 
alternative schools were also governed by the statute. Schools agreeing to admit 
voucher students were allowed to give first priority to current students and their 
siblings. Voucher applicants were then to be admitted by lottery until they made up 
20% of all the students in grades K-3. If a school still had vacancies it could then 
admit students whose families are members of organizations that provide financial 
support for the school. If there were still open seats, these were to be filled via a 
lottery in which all remaining applicants participated. For most existing schools this 
order of admissions priority means that voucher students will make up at most 20% 
of their populations. For schools with very large numbers of vacancies, including 
new schools that have no slots filled by current students and few applicants other 

1 The Ohio Supreme Court upheld a lower court's ruling that the voucher program did not violate 
Constitutional principles on separation of church and state. See Gatton et. al. vs. Goff, and Doris 
Simmons-Harris et. al. vs. Goff, Franklin County Curt of Common Pleas, Case No. 96-CVH-01-193, July 
21, 1996. 
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than voucher recipients, the proportion of voucher students enrolled could rise as 
high as 100%. 

The maximum value of a voucher for a student enrolling in a private school 
was to be $2,500, less 10% for a child whose family income was less than 200% of the 
federally-established poverty line, or less 25% for a child whose family income was 
more than twice the poverty line. Private schools with posted tuition rates less than 
$2,500 could not raise their rates for voucher students. Families were required to 
make arrangements for the tuition amounts not paid by the state (up to $250 in the 
case of those whose incomes were below twice the poverty line and up to $625 for 
those whose incomes exceeded twice the poverty line). Parents could work in the 
schools to make up the difference. Schools could also waive the remaining tuition. 
Children eligible for special education could participate in the program, and the 
Cleveland public schools would continue to pay the excess cost of special services 
allocated to cover those children's individualized education plans. 

If Cleveland voucher students enrolled in suburban public schools, the 
receiving school would be funded under both the state s open enrollment rules, 
which permit transfer of state funds amounting to approximately $3,300 for each 
student who transfers, and under the voucher program. Thus, they could receive as 
much as $5,550 - the state's normal funding plus $2,250, the maximum value of a 
voucher. 

Once admitted to the voucher program, students could continue receiving 
assistance through grade eight, provided the state legislature appropriated the 
necessary funds. Subject to appropriations, the program would continue 
indefinitely, each year admitting new children entering kindergarten through grade 
three. In any year the legislature failed to appropriate funds for the program, 
alternative schools would be allowed to charge former voucher recipients an 
amount limited to the value of the voucher plus the family's contribution. In late 
1996 it was not known whether a new cohort of voucher students would be 
admitted for the 1997-1998 school year. 

The cost of the vouchers was to be borne by the state, but out of special 
program funds, some of which might otherwise have gone to the Cleveland Public 
Schools. In the first year of the program, Cleveland public schools lost no money 
because students who transferred to alternative schools were still included in 
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Cleveland's count of average daily school membership. The state also reimbursed 
the Cleveland public schools for the cost of transporting voucher program students 
to the alternative schools. In subsequent years, however, voucher students would 
not be included in Cleveland's funding base. 

In the late summer of 1995, the Ohio Department of Education established a 
Cleveland Scholarship and Tutoring Office, (hereafter the Cleveland Office) which 
was responsible for receiving family scholarship applications, registering local 
private and suburban schools willing to accept voucher students, verifying students' 
family incomes and eligibility for the program, managing the admissions lottery 
required by law, and arranging student admissions and tuition payments. 2 The 
Cleveland Office solicited applications from parents and potential alternative 
schools, conducted a lottery as required by the statute, and arranged for the first 
group of voucher students to enter alternative schools in September, 1996. Because 
no neighboring public school district agreed to accept Cleveland voucher students, 
all participating schools were private schools operating within city limits. 3 

THE PROGRAM AS IMPLEMENTED 

The State Department of Public Instruction exercised policy discretion granted 
by the statute in several ways: it limited voucher eligibility to children whose family 
incomes were at or below 200% of poverty; it limited the proportion of scholarships 
for children previously attending private schools to 25%, not the 50% allowed in the 
statute 4 ; it conducted the lottery to give first priority to children whose families were 
below 100% of the poverty line; and it constructed the lottery to ensure that at least 
70% of the low income students selected were African American, so that the 
selection would reflect the current racial balance of the Cleveland public schools. 



2 The law required the state to make out tuition checks in the names of the recipient children's parents. 
To ensure that these checks were used for tuition and not general family income, checks were sent to the 
schools and endorsed over to the school by parents. The Ohio Courts cited this arrangement as evidence 
that state support went to families, not private schools. 

3 The Cleveland office apparently made serious efforts to recruit suburban schools to the program, but 
all refused. Reasons cited included limited vacancies, mismatch between existing programs and voucher 
students' needs, and suburban districts' unwillingness to participate in a program perceived as 
threatening to a neighboring public school system. 

4 The Cleveland Office considered all children entering kindergarten in fall 1996 to be public school 
students. Because kindergarten students were more than 1/4 of the total number of K-3 students in the 
voucher program, this allowed former private school students to make up 33% or more of the voucher 
students in grades 1 through 3. 
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Table 1 summarizes key differences among statutory provisions, state policy, and 
actual implementation. 

Table 1: Key Elements of the Program in State Law and Policy and as 
Implemented by the Cleveland Office 



Maximum % of 
vouchers that can 
go to students 
already attending 
private school 

Eligibility of 
Students with 
family income 
between 100% and 
200% of poverty 



Eligibility of 
students with 
family income 
>200% of poverty 

Proportion of 
voucher students 
who must be 
African American 



Alternative schools 
in which voucher 
winners can enroll 



Provisions of 
Statute 



Up to 50% 



Not addresses 



Eligible: voucher 
worth 75% of 
maximum $2500 
tuition 



No Provision 



Private schools in 
Cleveland and 
public schools in 
neighboring 
districts 



State Policy 



Up to 25% 



Defined as second 
priority in lottery 



Lower priority 



Same as in statute 



Program as 
Implemented 

As near 25% as 
possible, depending 
on number of 
public school 
voucher students 
enrolling 
(currently -29%) 

375 offered 
vouchers after all 
poverty families 
were given the 
chance to verify 
income 

None enrolled 



Only Cleveland 
private schools 
(due to suburban 
publics' refusal) 



70%, to reflect 
public schools' 
racial composition 
and abide by 
Federal Court 
Consent Decree 



70% of 

families offered 
vouchers 
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For the 1996-1997 school year, parents applied on behalf of 6,246 children for 
slots in private schools. 5 Since school tuition varied between $900 and $2,500, it was 
not known in advance how many vouchers would be created under the program. 

As it turned out, 1,800 vouchers were awarded at an average tuition cost of $1,800. 

Of the applications received, 1,790 were for students already attending private 
schools and 4,456 were for students either attending Cleveland Public schools or 
entering kindergarten. Since the number of slots for students already attending 
private schools was limited to 1/4 of the total, approximately 450 slots were available 
for students already attending private schools. 

The Cleveland Scholarship Office conducted a lottery in January, 1996. A total 
of 1125 low income public school students (790 African American) and 375 low 
income private school students (not divided by race) were drawn, plus a waiting list 
of 285 students drawn in the same proportions. Families of the low income 
students drawn in the lottery were offered the opportunity to arrange private school 
placements. These families were asked to visit the Cleveland Office to verify income 
and those that did were given their choice among the available private school slots. 
All other applicants were given lottery numbers and placed on secondary waiting 
lists. 



Because many of the public school students originally drawn did not take the 
option of enrolling in private schools, all the public school families on the original 
waiting list were asked to verify their incomes. All families that verified their 
incomes were eventually offered vouchers. In March, 1996, as it became obvious 
that some private school slots would still be available, all the public school families 
in the original applicant pool that had incomes below the poverty level and had not 
already been offered a voucher were offered opportunities to verify income against 
the possibility that they might be offered slots in private schools. Those families that 
appeared and verified their low income status were all offered vouchers. In June, 
1996, 375 applicants whose family incomes were between 100% and 200% of poverty 
were asked to verify income and those who did so were offered vouchers. After 
school opened in the fall, some vacancies in private schools still remained, and a 



5 All numbers cited in this paper are approximate: the Cleveland Office continues to refine its data on 
applicants, based on efforts to remove duplicate applications, verify family claims on income and prior 
school placements, and adjust for mid-year changes in student enrollment (i.e. entry of new students into 
the program and attrition from alternative schools). 
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total of 1,000 below-poverty income families were contacted again in November and 
given another opportunity to verify income and apply for remaining slots on a first- 
come first-served basis. If any private school slots were still available after 
November, they would be offered to non-poverty families. 



The only group of eligible applicants who were not eventually offered the 
opportunity to verify income and receive vouchers were members of the waiting 
list for children who had attended private schools in the previous year. This waiting 
list did not clear because families already paying private school tuition seldom 
turned the vouchers down. The numbers of slots for these students were also 
limited by the 25% rule set by state policy: since former public school students filled 
fewer slots than expected, fewer slots could be offered to private school students. 
Table 2 summarizes the sequence of offers made to groups of parents. 

Table 2: Public School Students' Opportunities to Receive Vouchers 





Offered Slots 
after Jan. 1996 
lottery 


Asked to verify 
income in 
February, 1996* 


Asked to verify Asked to verify 
income in income in 

March and November, 

June 1996* 1996** 


Income <100% 
poverty 


1125 


-660 


-1375 -1000 


Income <200% 
poverty 


0 


0 


952 0 


NB. There is some duplication in the counts in columns 2, 3, and 4 

* Families were asked to verify incomes in order to stay on the 



waiting list. All who verified incomes were offered vouchers. 
** Families were asked to verify incomes in order to gain access to 
private school slots on a first come-first served basis 



By late October, 1996 some 1801 voucher recipients were enrolled in a total of 
51 private elementary schools, 35 Catholic and all but 4 of the others affiliated with 
religious denominations. Two new private schools, both non-sectarian, were created 
in response to the voucher program. One of the new schools had the highest 
enrollment of voucher students, 258. Of the religious schools, the one enrolling the 
most students was a Christian school with 141 voucher recipients. Eight schools 
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enrolled less than ten voucher students. Kindergarten students accounted for 635 of 
the total; there were also 441 first graders, 404 second graders, and 321 third graders. 

PROGRAM ENROLLMENT PATTERNS AND EVALUATION DESIGN 

The Cleveland Office was able to fill 1801 slots from an original applicant list 
of 6,246. A total of 1,600 were excluded from the lottery because of income above 
200% of poverty, and an unknown number of additional applicants might have 
been ineligible, because they failed to provide information about family income. 

Approximately 4,000 families of public school students were ultimately made 
some kind of offer. The offers made to families differed from one time to another. 
The original lottery winners were told that they would be offered slots in private 
school, subject only to proof of income. Other groups were later asked to verify 
income in order to stay on a waiting list that had some chance of obtaining slots in 
private schools. Though all the families who verified income in response to the 
offers made in March, June, and November were eventually given the choice of 
enrolling their children in private school, families did not know that everyone who 
verified income would be offered a slot. Families that received the November offer 
faced a different situation -- they could claim vouchers on a first-come first-served 
basis, but they could enroll their children only in the private schools that still had 
vacancies. These families also faced a prospect that the original lottery winners did 
not, of transferring schools in mid-year. 

Why did so many offered vouchers not enroll, and what significance might 
the low enrollment rate have for future program management and evaluation? 

If voucher advocates and opponents agree on one thing, it is that parents who 
apply for vouchers are likely to use them. Most assume that parents who want to 
use vouchers either have strong complaints against the public schools or are 
strongly attracted to one or another characteristics of private schools. Consequently, 
the design of the Cleveland program did not anticipate a high refusal rate among 
the original lottery winners. Once the original lottery winners were processed, the 
Cleveland Office's offers to additional families were different and almost certainly 
less attractive. In retrospect a number of factors might, either separately or in 
combination, account for the high refusal rate, including: 
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• A federal court's lifting of a long-standing requirement for cross-district 
busing of elementary students. This occurred in May, 1996, long after 
parents had applied to the voucher program, and opened up an option 
within the public school system that had previously been unavailable. 

• The possibility that some families applied for vouchers because they 
expected a teachers' strike to close the public schools in September, 1996. 
Labor issues were settled and the strike averted before school opened. 

• The possibility that parents who won vouchers used them as leverage to 
negotiate desirable placements for their children in public schools, e.g. in 
magnet programs. 

• Parents' concerns over relatively long trips to private schools, using 
publicly-funded but unproved transportation arrangements. 

• The program's requirement that parents either pay or work for 

10% of their child's tuition (25% for families whose incomes were above 
200% of the poverty line). 

• Parents' possible reluctance to cooperate with the voucher program's 
income verification requirements. 

• Working families' reluctance to participate in a state benefit program that 
to them resembled welfare. 

• Controversy among African-American community leaders about the 
wisdom of abandoning public schools. 

• The possibility that some parents were interested only in enrolling their 
children in one of the well-known private schools, and finding those slots 
taken, preferred to keep children in public schools. 

• The controversy surrounding the voucher program, suggesting to some 
parents that tuition benefits might disappear after the first year. 

There are, in addition, some reasons to question whether all applications 
were bona fide. Parents whose original applications did not acknowledge all their 
income may have been reluctant to appear and have their incomes verified (which 
could be done by simply showing a pay stub for one parent or guardian). 



At this early stage in the life of the program it is impossible to know what the 
high refusal rate means about the demand for vouchers, parents' willingness to pay 
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small fractions of private school tuitions, or parents' views of the relative quality of 
private and public schools. With a year's experience, information networks are 
bound to develop, and parents might learn much more about what is available in 
private schools — they might conclude that previously unknown private schools are 
highly desirable, that transportation arrangements are acceptable, and that school 
activities in which some parents engage to pay the family's fraction of tuition are 
not burdensome. Parents who preferred the new option of enrolling their children 
in neighborhood public schools might also become disillusioned. These changes 
might increase the acceptance rate, should the program continue adding students in 
the future. Of course, the opposite could happen. Any strategy for evaluating the 
Cleveland Voucher Program must include a close study of parents' choices and 
satisfaction. 

The high refusal rate also has important implications for any effort to link 
participation in the voucher program to student outcomes. Because all the eligible 
students were eventually made some form of offer, and many did not respond to 
the request to verify income, there is no randomly chosen "control group" of lottery 
losers. Families that did not appear in response to requests to verify income might 
not have been eligible, or their desire to enroll their children in private schools 
might have been weak. The pool of low income families that did not ultimately 
enroll their children in private schools variously composed of several non- 
mutually exclusive groups, including those that rejected a sure voucher, those who 
knew their incomes exceeded the eligibility limits, those who were not interested in 
the private schools that still had vacancies after the original lottery winners were 
placed, and those who left Cleveland after applying for vouchers in January 1996. 

These facts eliminate the theoretically best approach to assessing program 
effectiveness; i.e., comparing the student outcomes (test score growth, attendance, 
course passing, interest in school) between two groups of students, namely (1) those 
attending private schools with voucher assistance and (2) those attending public 
schools after losing in a lottery. 6 This ideal approach allows comparisons between 

6 The widely publicized debate about evaluation of the Milwaukee voucher program has turned on the 
question of whether there was a randomly selected group of voucher "losers" that could be used as a 
control. In this case the argument is moot: there is no such group. Re. Milwaukee, see Peterson, Paul E., 
Jay P. Greene and Chad Noyes, School Choice in Milwaukee, The Public Interest, Fall 1996; also Witte, 
John F., Politics, Who Benefits from the Milwaukee Choice Program? in Richard Elmore, Bruce Fuller, 
and Gary Orfield, eds.. Who Chooses, Who Loses? Culture, Institutions, and the Unequal Effects of 




12 



10 



groups of children whose parents expressed interest in private school, some of who 
won a voucher lottery, and others of whom lost. Such a comparison eliminates 
"self-selection bias" — unmeasurable difference in attitudes and motivations 
between families that sought entry to private schools and those that did not. Because 
there is no randomly assigned group of "losers" in the voucher lottery, another and 
somewhat less desirable approach, comparing voucher students' outcomes with 
those of a statistically constructed sample of Cleveland public school students, is the 
only option available. 

A serious assessment of the Cleveland voucher program must therefore have 
at least two parts: a study of parents' choices and satisfaction, comparing those who 
declined vouchers and those who accepted them; and a statistical study of voucher 
students' outcomes relative to demographically similar Cleveland public school 
students. The next two sections will provide specifications for these studies. The 
conclusion will also include recommendations about how the State of Ohio can 
administer the voucher program in future years to ensure that comparisons 
between randomly selected groups of students can be made. 





School Choice. New York, Teachers College Press, 1996 
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CHOICE AND SATISFACTION STUDY 

The "Choice and Satisfaction" study will be conducted to achieve three 
goals: (1) understand why many families that applied for vouchers ultimately 
declined the opportunity to enroll children in private schools; (2) assess initial 
parental satisfaction with the program; and (3) understand the ways in which 
Cleveland private and public schools responded to the voucher program, by creating 
new vacancies and tailoring programs to appeal to voucher students. Offerors' work 
plan for this study must have the following components: 

1. Face-to-face interviews with parents (minimum 25) to develop a 

more complete list of the factors that contributed to their accepting versus rejecting 
the vouchers they were offered. These factors might include the reasons for rejecting 
vouchers listed above. The interviews should also explore the factors that affected 
parents' satisfaction with the voucher program overall and with their children's 
current school placement and experience. 

The parents interviewed for this purpose should be a stratified cross-section 
of all applicants, both those that eventually enrolled children in voucher schools 
and those that did not. Possible stratification factors for selecting parents include the 
child's grade and income level (above or below poverty level), neighborhood (zip 
code), whether the child was enrolled in a public or private school in the Spring of 
1996, whether the child was one of the initial lottery winners versus whether the 
child was initially placed on one or another waiting list, and whether the child is 
now using a voucher to attend private school. Information on all families that 
applied for vouchers is now available in machine readable form. By December 31, 
1996 the Cleveland Office will also provide in machine readable form data regarding 
the school preferences listed on the application forms. 

Interviews may be conducted in small groups or one-on-one at the Cleveland 
Office, at neighborhood schools or churches, or in other locations proposed by the 
study contractor. Parents should receive a small honorarium (e.g., $10.00) for 
participating in this activity. The project also might reimburse for taxi fares. 

2. Use of the information from Step #1 to develop and pilot test an easy to 
read, group administered, parent questionnaire. This questionnaire, which 
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should take no more than 30 minutes to complete, would inquire about why 
parents applied for the program, why they did or did not accept a voucher, 
and their satisfaction with the choices they made. 

3. Survey of approximately 500 parents who applied for the voucher 
program. Parents should be selected via a sampling plan using the types of 
stratification factors listed above. Parents selected for the survey may be contacted by 
a combination of mail, telephone, school announcements, and other forms of 
invitation. 

4. Administration of the questionnaire at a minimum of five sites in the 
Cleveland area. Sites may be public or private middle or high school 

cafeterias or auditoriums (or other school facilities in which seats are large enough 
to accommodate adults). Parents should receive a nominal honorarium (e.g., $10.00) 
and refreshments for participating in this activity. The project also might 
arrange busing or reimburse for taxi fares to further insure a high response rate and 
thereby avoid biasing results by including only the most intensely interested 
parents. 

A group administered survey is suggested because of concerns about 
the parents' reading proficiency, their access to private telephones at home, and 
related issues. Questionnaires must not be anonymous because it is necessary to 
link responses with data from the parents' voucher application forms (e.g. parents 
expression of preference among alternative schools) and with individualized 
demographic and test score data in the Cleveland Public School database. 

5. Analysis of the survey data and preparation of a report of the 

findings on bases for parents choices among schools and their satisfaction with their' 
children's current schooling experiences. This report should be suitable for 
submission to a professional journal; e.g., it should include a review of the 
literature regarding school choice. It also should contain suggestions for possible 
changes in how the program was implemented, including eligibility rules. 

6. Performance and analysis of a survey of heads of participating alternative 
schools and of Cleveland public schools from which large numbers of students left 
to enter alternative schools. The survey will explore the schools' supply responses. 
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In the case of private schools admitting voucher students it will assess schools' 
policies and methods regarding creating new vacancies for voucher students (e.g. 
expanding classrooms, hiring new teachers, or filling only previously empty slots), 
and creation of new instructional programs to attract or meet the needs of voucher 
students. It will also inquire about voucher students' failure and attrition rates, and 
instructional challenges such students pose in private schools. In the case of 
Cleveland public schools the survey will explore schools' efforts to attract voucher 
students who had the choice of attending private schools, and perceived effects of 
voucher students' departure on school enrollment, funding, and programs. 

Offerors should display understanding of the issues to be explored in this 
study, and provide detailed explanations of the sampling, interviewing, and analysis 
methods to be used. Offerors should devote particular attention to methods of 
gaining high response rates among low income families and conducting interviews 
with adults who might have low degrees of reading skill. The project schedule must 
also take account of the urgency of interviewing parents before their memories of 
choices made in early 1996 have time to fade. 

(The RFP should specify what resources and assistance the Cleveland Office 
can provide, including identifying places to conduct the surveys, inviting parents to 
participate, transportation arrangements, etc.). 
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INITIAL EFFECTIVENESS STUDY 



The primary goal of the Cleveland voucher program is to improve student 
achievement in reading and mathematics. However, as explained above, obtaining 
a valid measure of effectiveness will be difficult because of the lack of a control 
group of voucher applicants who "lost" in a lottery. Virtually every program 
eligible family that applied for a voucher was offered some form of opportunity that 
could have led to their child's enrollment in a private school. Many either turned 
down the opportunity or failed to take action that could have resulted in a voucher. 
Nevertheless, it may be possible to glean some information about initial program 
effects (i.e., after one year of operation) by examining whether voucher students 
score higher or lower than similarly situated non-voucher students. (If the program 
continues admitting new students after the 1996-1997 school year it might be 
possible to create a randomly selected control group for future cohorts of students. 

But the study described here is for the first-year cohort, for which no such control 
group can exist). 

The feasibility of an effectiveness study on the 1996-1997 cohort depends on 
several factors, not the least of which are (1) the ability to identify and read the 
machine readable records of 1996 Cleveland public school children who are 
currently enrolled in the voucher program and (2) the completeness of these 
records. Before the study is initiated it must be ascertained that public school records 
contain students' Spring 1996 reading and mathematics test scores, birth date, 
number of parents in the home, whether the student is or is not in the free lunch 
program, and other relevant demographic data. (Such data do not appear to exist for 
students who were enrolled in private schools in Spring 1996). 

If the public school records of those currently enrolled in the program can be - 
found and if they are reasonably complete, then some indication of program 
effectiveness can be obtained by comparing these data to scores students earn on a 
special (contractor conducted) May 1997 test administration. 

This administration would involve two groups of students. Group A 
would consist of all the program children who are currently in grades 2 and 3 at 
public and private schools and whose 1995-96 machine readable records could be 
found. Group B would consist of current second and third graders at Cleveland 
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public schools who are not in the voucher program and whose 1995-96 machine 
readable records could be found. 

A regression equation could be constructed with group B. This equation 
would use the spring 1996 test scores and background data to estimate a 
student's spring 1997 scores. This equation could then be applied to the 
students in Group A to see if their spring 1997 scores are generally 
higher, lower, or about the same as would be "expected" (i.e., based on 
the typical relationships among these variables in Group B). To enhance 
the credibility of this study, the spring 1997 testing should be done in 
both public and private schools, it should be independent of any other 
testing that was being done in these schools, and the tests should be 
administered by the contractor (although a classroom teacher also would be 
present). The test used for this purpose should not be identical to the 
ones used by the district (but an alternate form may be satisfactory). 

Offerors may suggest one or more alternative strategies for measuring 
program effectiveness, including forms of tests to be used, sampling of schools and 
students, methods of test administration, and data analysis. Offerors must describe 
their methods for conducting the study described above but they may also propose 
alternative approaches. For each proposed approach, the contractor should discuss 
its advantages and limitations for assessing the program's ability to improve student 
reading and mathematics scores. This discussion should include consideration of 
such issues as attrition from the program, voucher and non-voucher 
students changing schools during the year, and related concerns that could 
threaten or impact the internal or external validity of the study. 

Offerors should provide a detailed data collection and analysis plan. This plan' 
should describe the number of students tested, the measures that will be used, and a 
discussion of the analytic methods employed. The contractor should also identify 
supplementary information about students to be collected from the school, e.g. 
attendance rates, grades, course placement and passing, etc. The following 
information is provided to help the contractor develop this plan: (To be provided in 
the RFP by the Ohio Department of Education). 

There are currently 1801 students in the Cleveland voucher program. 
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Tables show the approximate number of voucher children at each grade 

level that are currently in public and private schools who did and did not 
come from the Cleveland city schools. 

Table shows the number of voucher children by grade level at each 

school as of 10/31/96 broken down by whether or not their machine readable 
Cleveland city public school records could be found. 

Table shows the number of voucher students (by grade level and 

variable) whose data could be found in the Cleveland city public school 
machine readable database. 

Table shows the correlation among selected variables in the Cleveland 

city public school machine readable database. These variables include the 
students' reading and mathematics test scores in grades 1, 2, and 3; 
number of parents in the home, whether receiving free lunch, etc. 
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RECOMMENDATIONS FOR FUTURE VOUCHER PROGRAM ADMINISTRATION 

Experience with voucher program admissions in 1996 suggests that creation 
of a randomly selected control group of lottery "losers" is not automatic, and that no 
such group will be available unless program managers take special care. This section 
suggests actions that might make it possible for the outcomes of future cohorts of 
Cleveland voucher students to be compared with a truly similar group of students 
attending the Cleveland public schools. These suggestions apply only if additional 
students are admitted for 1997-1998 or subsequent years. They also assume that new 
students are admitted for some grades other than kindergarten. Families seeking 
vouchers for kindergarten will include some that would have enrolled their 
children in private schools in any case. Thus, a number of "losers" are likely to 
attend the same private schools as "winners," confounding any comparisons of 
winners' and losers' school outcomes. 

The Problem 

If there is a high rejection rate among voucher applicants chosen by lottery, it 
is extremely difficult to create a randomly selected losers' pool. The families that are 
offered vouchers and turn them down are obviously different from those who 
accept; the losers' pool would almost certainly include families that would have 
turned down vouchers if they were offered. Thus, when acceptance rates are 
significantly less than 100% (e.g. 90% or less) the pool of "loser" applicants will 
always be very different from the pool of those using vouchers to enroll in private 
schools. 

Implications 

The Cleveland Scholarship Program's experience with relatively high refusal 
rates suggests the need to pre-screen applicants in an effort to cut down the number 
who would not take vouchers even if they won. Such a pre-screen must eliminate 
applicants whose family incomes make their children ineligible, families that do not 
live in Cleveland, and families that might falsely claim their children attended 
public school in the previous year. At the time of their first application, parents 
should be required to: 

Demonstrate income with something like a tax return or welfare form, not 

just a pay stub from one job. 
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Provide conclusive proof of residence in Cleveland. 

Provide documentary evidence of what school their child is currently 

attending. 

Give detailed information about the child: full name, birth date, schools 

previously attended, and social security number if available. 

The family application form should also include a place for a work contact for 
one parent and a second contact for the family -- another family member, friend, or 
pastor, who will know how to reach the parents if the family moves. All enrollment 
information should be computerized immediately. 

Parents should also be pre-screened to eliminate those likely to reject a 
private school placement if offered. In particular, parents should be required to sign 
a form saying they understand their child's attendance at private school could cost 
them as much as $250 /yr, in tuition, or that they might have to work in the school 
in lieu of paying their share of tuition. The program should also provide a great 
deal of advance information about private school options before, not after, families 
a PPly, v * a brochures, community meetings, school fairs, school tours, etc. If possible, 
parents should be asked before the lottery whether there is only a limited set of 
private schools to which they would send their children in preference to the 
available public schools. 

Between the time families apply and are entered into a lottery, the Cleveland 
Office should also obtain student numbers for those currently in public schools. This 
should be entered into the student's computerized record, as should any forms used 
to register school preferences and choices. 

The lottery might also be administered to minimize parental refusals. If 
parent preference forms indicate that some private schools are over-subscribed, 
lotteries for admissions to those schools should be conducted first. Parents who 
indicate that they will send their children only to those private schools, and who 
lose in the lottery, could then be considered part of a randomly-selected "losers' 
pool." 
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After the lottery, the key to maximizing the voucher acceptance rate is 
prompt and insistent notification of families. The program should inform parents 
through multiple channels -- home addresses and telephone numbers, work 
addresses, secondary family contacts, and current school placements. 

Once students are enrolled in school, the program needs to stay in contact 
with as many students as possible, both lottery winners and losers. High attrition 
from either group will seriously threaten the validity of study findings. Given the 
high turnover rates in urban schools, effectiveness studies are constantly threatened 
by this problem; it is particularly serious if one group (likely the students attending 
public schools) suffers attrition at a far higher rate than the group to which it is 
compared. Preventing devastating sample attrition will require frequent (possibly 
twice a year) checks of both public and private schools to verify current school 
placements and obtain any available information on students who have departed 
the school. The Cleveland office or a contractor hired to conduct effectiveness 
studies should also promptly follow up family contacts to find students who have 
changed schools. 

These measures are particularly critical for any multi-year study of a given 
cohort of students. The state should be aware that any effort to conduct such a study 
could be defeated by sample attrition, no matter how diligently the state and 
contractor work to maintain contact with students. 
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